A Dataset of Multi-Illumination Images in the Wild

Collections of images under a single, uncontrolled illumination have enabled the rapid advancement of core computer vision tasks like classification, detection, and segmentation. But even with modern learning techniques, many inverse problems involving lighting and material understanding remain too severely ill-posed to be solved with single-illumination datasets. To fill this gap, we introduce a new multi-illumination dataset of more than 1000 real scenes, each captured under 25 lighting conditions. We demonstrate the richness of this dataset by training state-of-the-art models for three challenging applications: single-image illumination estimation, image relighting, and mixed-illuminant white balance.Comment: ICCV 201

Computational Mirrors: Blind Inverse Light Transport by Deep Matrix Factorization

We recover a video of the motion taking place in a hidden scene by observing changes in indirect illumination in a nearby uncalibrated visible region. We solve this problem by factoring the observed video into a matrix product between the unknown hidden scene video and an unknown light transport matrix. This task is extremely ill-posed, as any non-negative factorization will satisfy the data. Inspired by recent work on the Deep Image Prior, we parameterize the factor matrices using randomly initialized convolutional neural networks trained in a one-off manner, and show that this results in decompositions that reflect the true motion in the hidden scene.Comment: 14 pages, 5 figures, Advances in Neural Information Processing Systems 201

Computational bounce flash for indoor portraits

Thesis: S.M., Massachusetts Institute of Technology, Department of Electrical Engineering and Computer Science, 2017.Cataloged from PDF version of thesis.Includes bibliographical references (pages 31-33).Portraits taken with direct flash look harsh and unflattering because the light source comes from a small set of angles very close to the camera. Advanced photographers address this problem by using bounce flash, a technique where the flash is directed towards other surfaces in the room, creating a larger, virtual light source that can be cast from different directions to provide better shading variation for 3D modeling. However, finding the right direction to point a bounce flash towards requires skill and careful consideration of the available surfaces and subject configuration. Inspired by the impact of automation for exposure, focus and flash metering, we automate control of the flash direction for bounce illumination. We first identify criteria for evaluating flash directions, based on established photography literature, and relate these criteria to the color and geometry of a scene. We augment a camera with servomotors to rotate the flash head, and additional sensors (a fisheye and 3D sensors) to gather information about potential bounce surfaces. We present a simple numerical optimization criterion that finds directions for the flash that consistently yield compelling illumination and demonstrate the effectiveness of our various criteria in common photographic configurations.by Lukas Murmann.S.M

Computational illumination for portrait photography and inverse graphics

Supervised training of deep networks has led to remarkable successes in computer vision, for example on image classification or object detection problems. These successes are driven by the availability of large amounts of paired training data with manual ground truth annotations. For many photography or inverse graphics applications however, manual annotation of ground truth labels is not viable. Motivated by this, the research presented in this thesis proposes several portable hardware prototypes that enable the collection of training data for applications ranging from non-line-of-sight imaging to relighting and dark-flash photography. The thesis also discusses a novel formulation for fast and accurate differentiable rendering based on analytical anti-aliasing. It is demonstrated how this renderer can be used for inverse graphics problems. The thesis concludes with a discussion on how differentiable programming can be combinded with data-driven feed forward networks for practicle inverse graphics applications.Ph.D

NFC Heroes - Observing NFC Adoption through a Mobile Trading Card Game

Near-field Communication (NFC) technology finally starts to proliferate on modern smartphones, enabling researchers to conduct researchin the real world. The research question for this work is to learn about the distribution of NFC tags in the wild. As there is, for good and for bad, no central registry or database of NFC tags, we propose a game-based approach to capture the adoption of NFC solutions and technologies.We first report on the development process an NFC-based game. We then present the game logic and implementation, share our experiences from two release cycles on Googles Play Store and finally report on initial results and lessons learned during the whole process.Godkänd; 2012; 20121012 (matkra

Computational bounce flash for indoor portraits

Portraits taken with direct flash look harsh and unflattering because the light source comes from a small set of angles very close to the camera. Advanced photographers address this problem by using bounce flash, a technique where the flash is directed towards other surfaces in the room, creating a larger, virtual light source that can be cast from different directions to provide better shading variation for 3D modeling. However, finding the right direction to point a bounce flash requires skill and careful consideration of the available surfaces and subject configuration. Inspired by the impact of automation for exposure, focus and flash metering, we automate control of the flash direction for bounce illumination. We first identify criteria for evaluating flash directions, based on established photography literature, and relate these criteria to the color and geometry of a scene. We augment a camera with servomotors to rotate the flash head, and additional sensors (a fisheye and 3D sensors) to gather information about potential bounce surfaces. We present a simple numerical optimization criterion that finds directions for the flash that consistently yield compelling illumination and demonstrate the effectiveness of our various criteria in common photographic configurations

Computational mirrors: Blind inverse light transport by deep matrix factorization

We recover a video of the motion taking place in a hidden scene by observing changes in indirect illumination in a nearby uncalibrated visible region. We solve this problem by factoring the observed video into a matrix product between the unknown hidden scene video and an unknown light transport matrix. This task is extremely ill-posed as any non-negative factorization will satisfy the data. Inspired by recent work on the Deep Image Prior, we parameterize the factor matrices using randomly initialized convolutional neural networks trained in a one-off manner, and show that this results in decompositions that reflect the true motion in the hidden scene.United States. Defense Advanced Research Projects Agency (Contract HR0011-16-C-0030)National Science Foundation (U.S.) (Grant CCF-1816209